On the Effect of Inter-observer Variability for a Reliable Estimation of Uncertainty of Medical Image Segmentation

Uncertainty estimation methods are expected to improve the understanding and quality of computer-assisted methods used in medical applications (e.g., neurosurgical interventions, radiotherapy planning), where automated medical image segmentation is crucial. In supervised machine learning, a common practice to generate ground truth label data is to merge observer annotations. However, as many medical image tasks show a high inter-observer variability resulting from factors such as image quality, different levels of user expertise and domain knowledge, little is known as to how inter-observer variability and commonly used fusion methods affect the estimation of uncertainty of automated image segmentation. In this paper we analyze the effect of common image label fusion techniques on uncertainty estimation, and propose to learn the uncertainty among observers. The results highlight the negative effect of fusion methods applied in deep learning, to obtain reliable estimates of segmentation uncertainty. Additionally, we show that the learned observers' uncertainty can be combined with current standard Monte Carlo dropout Bayesian neural networks to characterize uncertainty of model's parameters.Comment: Appears in Medical Image Computing and Computer Assisted Interventions (MICCAI), 201

Uncertainty-driven Sanity Check: Application to Postoperative Brain Tumor Cavity Segmentation

Uncertainty estimates of modern neuronal networks provide additional information next to the computed predictions and are thus expected to improve the understanding of the underlying model. Reliable uncertainties are particularly interesting for safety-critical computer-assisted applications in medicine, e.g., neurosurgical interventions and radiotherapy planning. We propose an uncertainty-driven sanity check for the identification of segmentation results that need particular expert review. Our method uses a fully-convolutional neural network and computes uncertainty estimates by the principle of Monte Carlo dropout. We evaluate the performance of the proposed method on a clinical dataset with 30 postoperative brain tumor images. The method can segment the highly inhomogeneous resection cavities accurately (Dice coefficients 0.792 $\pm$ 0.154). Furthermore, the proposed sanity check is able to detect the worst segmentation and three out of the four outliers. The results highlight the potential of using the additional information from the model's parameter uncertainty to validate the segmentation performance of a deep learning model.Comment: Appears in Medical Imaging with Deep Learning (MIDL), 201

Impact of random outliers in auto-segmented targets on radiotherapy treatment plans for glioblastoma.

AIMS To save time and have more consistent contours, fully automatic segmentation of targets and organs at risk (OAR) is a valuable asset in radiotherapy. Though current deep learning (DL) based models are on par with manual contouring, they are not perfect and typical errors, as false positives, occur frequently and unpredictably. While it is possible to solve this for OARs, it is far from straightforward for target structures. In order to tackle this problem, in this study, we analyzed the occurrence and the possible dose effects of automated delineation outliers. METHODS First, a set of controlled experiments on synthetically generated outliers on the CT of a glioblastoma (GBM) patient was performed. We analyzed the dosimetric impact on outliers with different location, shape, absolute size and relative size to the main target, resulting in 61 simulated scenarios. Second, multiple segmentation models where trained on a U-Net network based on 80 training sets consisting of GBM cases with annotated gross tumor volume (GTV) and edema structures. On 20 test cases, 5 different trained models and a majority voting method were used to predict the GTV and edema. The amount of outliers on the predictions were determined, as well as their size and distance from the actual target. RESULTS We found that plans containing outliers result in an increased dose to healthy brain tissue. The extent of the dose effect is dependent on the relative size, location and the distance to the main targets and involved OARs. Generally, the larger the absolute outlier volume and the distance to the target the higher the potential dose effect. For 120 predicted GTV and edema structures, we found 1887 outliers. After construction of the planning treatment volume (PTV), 137 outliers remained with a mean distance to the target of 38.5 ± 5.0 mm and a mean size of 1010.8 ± 95.6 mm3. We also found that majority voting of DL results is capable to reduce outliers. CONCLUSIONS This study shows that there is a severe risk of false positive outliers in current DL predictions of target structures. Additionally, these errors will have an evident detrimental impact on the dose and therefore could affect treatment outcome

Deep-Learning-Based Dose Predictor for Glioblastoma–Assessing the Sensitivity and Robustness for Dose Awareness in Contouring

External beam radiation therapy requires a sophisticated and laborious planning procedure. To improve the efficiency and quality of this procedure, machine-learning models that predict these dose distributions were introduced. The most recent dose prediction models are based on deep-learning architectures called 3D U-Nets that give good approximations of the dose in 3D almost instantly. Our purpose was to train such a 3D dose prediction model for glioblastoma VMAT treatment and test its robustness and sensitivity for the purpose of quality assurance of automatic contouring. From a cohort of 125 glioblastoma (GBM) patients, VMAT plans were created according to a clinical protocol. The initial model was trained on a cascaded 3D U-Net. A total of 60 cases were used for training, 15 for validation and 20 for testing. The prediction model was tested for sensitivity to dose changes when subject to realistic contour variations. Additionally, the model was tested for robustness by exposing it to a worst-case test set containing out-of-distribution cases. The initially trained prediction model had a dose score of 0.94 Gy and a mean DVH (dose volume histograms) score for all structures of 1.95 Gy. In terms of sensitivity, the model was able to predict the dose changes that occurred due to the contour variations with a mean error of 1.38 Gy. We obtained a 3D VMAT dose prediction model for GBM with limited data, providing good sensitivity to realistic contour variations. We tested and improved the model’s robustness by targeted updates to the training set, making it a useful technique for introducing dose awareness in the contouring evaluation and quality assurance process

Chemoradiation of pancreatic carcinoma

Pancreatic carcinoma is a malignancy with a poor prognosis and the 4th. most common cause of cancer-related deaths. Patients are usually diagnosed at advanced stage of the disease. Surgical resection remains the only potentially curative therapy, as only 20% of the patients present with disease are amenable to resection. Surgery, chemotherapy, radiotherapy and palliative therapies are therapeutic options. Multidisciplinary approach is needed for every stage of the disease. Researches showed an improved survival benefit of radiotherapy (RT) and chemotherapy (CT) combination for locally advanced unresectable pancreatic carcinoma compared to RT or CT alone. In an attempt to improve survival, the efficacy of chemoradiation (CRT) after surgery compared to observation has been tested in several trials. Neoadjuvant CRT achieves a higher probability of margin negative R0 resection. Currently, both 5-FU and gemcitabine have been used concurrently with RT, and also targeted agents (erlotinib, cetuximab, panitumumab, bevacizumab) have been also evaluated

Radiosurgery and radiotherapy for arteriovenous malformations: outcome predictors and review of the literature.

Arteriovenous malformations (AVMs) are rare congenital vascular pathologies. The reported overall annual hemorrhage rate is 3.0%, for unruptured AVMs it is 2.2%, and for ruptured AVMs, 4.5%. The main goal of AVM treatment is to prevent intracerebral hemorrhage. This is achieved by complete nidus eradication. Interventional treatment options include microsurgery, embolization and radiosurgery, as well as multimodal approaches. Radiosurgery is a safe and effective alternative to surgery or embolization, especially for AVMs located in deep or eloquent brain regions, where invasive treatment cannot be performed. With the introduction of the Leksell Gamma Knife, AVMs became one of the most common indications for radiosurgical interventions (nearly 30% of the first 15-year experience). The current review discusses the role of radiosurgery in the treatment of AVMs, with a focus on outcome predictors and a discussion of the relevant literature

Uncertainty-driven Sanity Check: Application to Postoperative Brain Tumor Cavity Segmentation

Uncertainty estimates of modern neuronal networks provide additional information next to the computed predictions and are thus expected to improve the understanding of the underlying model. Reliable uncertainties are particularly interesting for safety-critical computer-assisted applications in medicine, e.g., neurosurgical interventions and radiotherapy planning. We propose an uncertainty-driven sanity check for the identification of segmentation results that need particular expert review. Our method uses a fully-convolutional neural network and computes uncertainty estimates by the principle of Monte Carlo dropout. We evaluate the performance of the proposed method on a clinical dataset with 30 postoperative brain tumor images. The method can segment the highly inhomogeneous resection cavities accurately (Dice coefficients 0.792 ± 0.154). Furthermore, the proposed sanity check is able to detect the worst segmentation and three out of the four outliers. The results highlight the potential of using the additional information from the model's parameter uncertainty to validate the segmentation performance of a deep learning model

On the Effect of Inter-observer Variability for a Reliable Estimation of Uncertainty of Medical Image Segmentation

Uncertainty estimation methods are expected to improve the understanding and quality of computer-assisted methods used in medical applications (e.g., neurosurgical interventions, radiotherapy planning), where automated medical image segmentation is crucial. In supervised machine learning, a common practice to generate ground truth label data is to merge observer annotations. However, as many medical image tasks show a high inter-observer variability resulting from factors such as image quality, different levels of user expertise and domain knowledge, little is known as to how inter-observer variability and commonly used fusion methods affect the estimation of uncertainty of automated image segmentation. In this paper we analyze the effect of common image label fusion techniques on uncertainty estimation, and propose to learn the uncertainty among observers. The results highlight the negative effect of fusion methods applied in deep learning, to obtain reliable estimates of segmentation uncertainty. Additionally, we show that the learned observers’ uncertainty can be combined with current standard Monte Carlo dropout Bayesian neural networks to characterize uncertainty of model’s parameters

Response assessment after stereotactic body radiation therapy for spine and non-spine bone metastases: results from a single institutional study.

BACKGROUND The use of stereotactic body radiation therapy (SBRT) for tumor and pain control in patients with bone metastases is increasing. We report response assessment after bone SBRT using radiological changes through time and clinical examination of patients. METHODS We analyzed retrospectively oligo-metastatic/progressive patients with bony lesions treated with SBRT between 12/2008 and 10/2018, without in-field re-irradiation, in our institution. Radiological data were obtained from imaging modalities used for SBRT planning and follow-up purposes in picture archiving and communication system and assessed by two independent radiologists blind to the time of treatment. Several radiological changes were described. Radiographic response assessment was classified according to University of Texas MD Anderson Cancer Center criteria. Pain response and the neurological deficit were captured before and at least 6 months after SBRT. RESULTS A total of 35 of the 74 reviewed patients were eligible, presenting 43 bone metastases, with 51.2% (n = 22) located in the vertebral column. Median age at the time of SBRT was 66 years (range 38-84) and 77.1% (n = 27) were male. Histology was mainly prostate (51.4%, n = 18) and breast cancer (14.3%, n = 5). Median total radiation dose delivered was 24 Gy (range 24-42), in three fractions (range 2-7), prescribed to 70-90% isodose-line. After a median follow-up of 1.8 years (range < 1-8.2) for survivors, complete or partial response, stable, and progressive disease occurred in 0%, 11.4% (n = 4), 68.6% (n = 24), and 20.0% (n = 7) of the patients, respectively. Twenty patients (57.1%) died during the follow-up time, all from disease progression, yet 70% (n = 14) from this population with local stable disease after SBRT. From patients who were symptomatic and available for follow-up, almost half (44.4%) reported pain reduction after SBRT. CONCLUSIONS Eighty percent of the patients showed local control after SBRT for bone metastases. Pain response was favorable. For more accurate response assessment, comparing current imaging modalities with advanced imaging techniques such as functional MRI and PET/CT, in a prospective and standardized way is warranted. Trial registration Retrospectively registered